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I  Executive  Summary 

Research  on  Building  Education  &  Workforce  Capacity  in  Systems  Engineering, 
(referred  to  as  the  SE  Capstone  Project),  is  conducting  research  to  understand  the 
methods  through  which  systems  engineering  learning  and  career  interest  may  be 
increased  among  undergraduate  and  graduate  engineering  students.  This  research  is 
being  conducted  in  the  context  of  14  “capstone”  courses,  in  most  cases  as  an 
integrative  culminating,  project-based  course  involving  teams  of  students  working 
together  on  the  development  of  a  product  or  prototype  that  addresses  a  real 
Department  of  Defense  need.  Implemented  as  pilot  courses  in  eight  civilian  and  six 
military  universities,  these  14  institutions  are  piloting  methods,  materials,  and 
approaches  to  create  new  courses  or  enhance  existing  courses  to  embed,  infuse,  and 
augment  systems  engineering  knowledge,  as  defined  by  the  SPRDE-SE/PSE 
Competency  Model,  among  undergraduate  and  graduate  students.  This  report  is  a 
snapshot  of  progress  and  preliminary  findings  as  reported  in  SE  Capstone  partners’ 
interim  reports  as  of  January  201 1  which,  in  most  cases,  represents  the  midpoint  of 
their  course  implementation. 

Each  university  chose  one  or  more  problem  areas  based  on  existing  faculty  expertise 
and  interest.  Two  civilian  universities  and  one  service  academy  chose  to  work  in  more 
than  one  of  the  DoD  problem  areas.  More  than  half  the  projects  (8)  addressed  DoD 
Problem  Area  1 ,  with  Problem  Area  4  as  the  next  most  commonly  addressed  topic,  with 
the  remaining  two  problem  areas  divided  among  the  other  institutions. 

The  key  features  that  differentiated  the  organizational  structure  of  the  programs  at  the 
different  Institutions  of  Higher  Education  (IHEs)  were  the  following: 

•  Faculty:  The  collaboration  of  two  or  multiple  faculty  members  on  course  design  and 
implementation.  At  1 1  institutions,  faculty  came  from  at  least  three  separate 
engineering  disciplines,  literally  embodying  the  multi-disciplinarity  of  a  systems 
engineering  team.  SE  faculty  were  the  largest  percentage  of  participating  faculty. 

•  Courses:  The  integration  of  the  SE  component  into  existing  courses  or  the  creation 
of  entirely  new  courses. 

•  Course  sequencing:  The  implementation  of  a  course  sequence  that  included  an 
introductory  course  followed  by  a  capstone  experience  or  a  capstone  experience 
only. 

•  Student  population:  The  involvement  of  either  undergraduates  and  graduate 
students  as  learners*  or  only  one  of  these. 


Some  institutions  engaged  graduate  students  as  teaching  or  technical  assistants,  but  here  we  are  referring  to 
graduate  students  who  are  participating  in  the  course  for  credit. 
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•  Mentors:  The  presence  and  level  of  active  and  meaningful  involvement  of  DoD  and 
industry  mentors  in  a  variety  of  student  learning  experiences. 

There  was  a  diverse  array  of  methods,  approaches  and  structures  for  the 
implementation  of  the  courses.  All  but  one  institution  integrated  the  RT-19  effort  into 
existing  SE  courses.  Thirteen  of  the  14  projects  implemented  the  RT-19  pilot  over  two 
semesters.  One  institution  offered  a  single-semester  capstone  project  course,  and  one 
conducted  a  one-semester  course  in  the  fall  and  spring  semesters.  Student  teams 
ranged  from  four  to  seven  members.  Nine  of  the  institutions  had  only  undergraduates  as 
students,  two  had  exclusively  graduate  students  and  three  had  both.  Class  sizes  varied 
widely  across  institutions,  ranging  from  a  low  of  3  or  4  students  to  48.  The  average 
class  size  was  20. 

The  most  prevalent  engineering  discipline  (among  students)  across  all  institutions  was 
Systems  Engineering,  followed  by  Mechanical  Engineering,  Electrical  Engineering, 
Computer  Science/Software  Engineering,  and  Industrial  Engineering.  Overall,  the 
student  population  was  over  three-quarters  male,  with  a  small  female  population  and  a 
small  percentage  of  students  selecting  not  to  report  their  gender.  Students’  reported 
ethnicities  included  64  percent  white;  1 1  percent  Asian;  7  percent  African  American;  5 
percent  Hispanic/Latino;  and  Hawaiian  or  other  Pacific  Islander  at  <1  percent  and 
unreported,  1 1  percent.  Approximately  40  percent  of  the  students  who  returned  surveys 
reported  no  experience  with  Systems  Engineering  and  another  20  percent  reported  that 
they  did  not  know  if  they  had  any  such  experience — for  a  total  of  60  percent  of  the  total 
population.  Of  those  students  who  did  report  previous  systems  engineering  experience 
(multiple  responses  were  allowed),  25  percent  had  taken  coursework,  12.5  percent  had 
had  full-time  employment  in  systems  engineering,  7  percent  had  a  coop  or  internship 
experience,  and  1  percent  had  systems  engineering  work  experience  during  the 
academic  year. 

Over  half  (52  percent)  of  all  students  reported  a  level  of  high  interest  in  becoming  a 
systems  engineer,  18  percent  reported  moderate  interest,  and  1  percent  reported  little 
interest.  Forty-two  percent  of  the  students  reported  that  they  might  want  to  work  for  the 
government  as  a  systems  engineer. 

Projects  with  DoD  mentors  reported  very  different  levels  of  interaction.  Over  46  percent 
of  Pis  reported  that  their  DoD  mentors  were  “very  involved;”  another  46  percent 
reported  not  having  a  mentor  or  not  yet  working  with  their  mentor.  DoD  mentors 
facilitated  student  learning  in  a  variety  of  ways;  however,  their  roles  differed 
instructional^  depending  on  whether  they  also  served  as  clients,  as  they  did  for  50 
percent  of  participating  institutions. 

Industry  mentors  took  on  roles  similar  to  the  roles  played  by  DoD  mentors — as  clients, 
consultants,  or  SMEs.  Pis  at  two  institutions  that  had  both  industry  and  DoD  mentors 
reported  that  having  both  types  of  mentor  benefited  the  students’  projects. 
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It  is  premature  at  this  stage  of  course  implementation  to  correlate  student  outcomes 
with  the  structure  or  content  of  the  courses  or  with  any  particular  strategies  or  course 
materials.  However,  the  Pis’  interim  reports  provide  some  insights  into  preliminary 
lessons  learned  about  course  objectives  and  implementation.  These  observations  fall 
into  several  categories: 

1 .  Challenges  teaching  the  broad  topic  of  systems  engineering  to  non-SE  majors  under 
time  constraints. 

2.  Challenges  with  equivalent  grading  policies  in  multi-disciplinary  teams,  particularly 
where  SE  was  an  overlay  to  an  existing  multi-disciplinary  team  structure. 

3.  Challenges  with  content-domain-specific  problem  areas  and  with  finding  meaningful 
ways  for  other  disciplinary  majors  to  contribute. 

4.  Motivating  external  mentors  to  bring  authentic  professional  experiences  to  the 
learning  experience. 

5.  SE  content  modules  provide  opportunities  to  bring  non-SE  majors  to  a  common 
understanding.  These  have  been  implemented  with  varying  frequency,  durations, 
and  numbers  across  several  projects. 

6.  Efforts  to  provide  specific  disciplinary  expertise  (internal  and  external)  to  infuse 
sufficient  content  knowledge  into  student  teams  such  that  students  are  able  to  focus 
on  the  bigger  SE  competencies. 

It  is  also  not  clear  at  this  stage  the  extent  to  which  RT-1 9  funding  has  created  entirely 
new  materials  or  simply  (and  in  some  cases,  substantially)  enhanced  existing  courses. 
This  is  an  area  for  further  investigation. 

Recommendations  that  emerge  from  a  review  of  the  implementing  institutions’  interim 
reports  suggest  that: 

1 .  More  planning  time  is  needed  to  effectively  plan  and  coordinate  course  materials 
and  assessments;  make  optimal  use  of  external  resources  such  as  DoD  and 
industry  mentors,  as  well  as  facilities  visits;  and  to  secure  buy-in  and  define  roles 
and  responsibilities  of  interdisciplinary  faculty  participants. 

2.  In  order  to  effectively  recruit  students,  the  specific  DoD  problem  area  should  be 
defined  and  disseminated  to  students  at  least  at  the  time  students  register  for  their 
next  cycle  of  courses  (the  term  prior  to  the  course  implementation). 

3.  Better  connections,  more  clearly  defined  roles,  and  stronger  commitment  by  DoD 
and  industry  mentors  would  enhance  students’  experiences  in  cases  where  the 
participation  of  external  mentors  has  been  lacking.  Consideration  of  a  nominal 
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financial  commitment  by  external  clients  may  increase  the  investment/commitment 
by  these  mentors. 

4.  A  list  of  required  and  recommended  faculty  events  and  student  programs  should  be 
made  available  to  Pis  to  encourage  maximum  participation  and  allocation  of 
sufficient  financial  resources. 

This  report  contains  a  snapshot  of  the  richness  of  the  14  SE  Capstone  courses  that 
have  been  and  are  being  implemented  at  14  institutions.  At  this  point  in  project 
implementation,  it  is  not  possible  to  provide  an  analysis  of  student  learning  outcomes. 
The  final  report  will  aim  to  connect  the  course  content  and  organization,  including 
materials  created  by  faculty  as  well  as  the  contributions  of  external  mentors,  with 
impacts  on  student  learning  of  SE  content,  their  interest  in  SE  careers,  and  their  interest 
in  DoD  problem  areas  and  careers. 
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Introduction 


A  45%  increase  is  expected  in  systems  engineering  (SE)  jobs  in  the  next  decade1  and 
there  have  been  numerous  studies  and  workshops  that  have  highlighted  the  shortfalls  in 
both  the  number  and  capability  of  the  SE  workforce.  The  July  2006  National  Defense 
Industrial  Association  (NDIA)  task  Force  noted  among  the  top  five  systems  engineering 
issues  the  lack  of  adequate,  qualified  SE  human  capital  resources  within  Government 
and  industry  for  allocation  on  major  programs.2  A  July  2010  NDIA  white  paper  on  critical 
systems  engineering  challenges  identified  Issue  2  as:  The  quantity  and  quality  of 
systems  engineering  expertise  is  insufficient  to  meet  the  demands  of  the  government 
and  defense  industry,  and  further  outlined  certain  recommendations  to  build  SE 
expertise  and  capacity.  In  particular,  it  recommended  developing  SE  expertise  through 
“role  definition,  selection,  training,  career  incentives,  and  broadening  ‘systems  thinking’ 
into  other  disciplines,”  and  made  a  number  of  specific  recommendations: 

b.  Continue  to  work  with  major  universities  and  professional  societies  to  add  an 
introductory  course  in  SE  in  all  undergraduate  engineering  and  technical 
management  degree  programs  (task  for  NDIA  SED  Education  and  Training 
(E&T)  Committee). 

c.  Work  with  major  universities  and  recommend  SE  curricula  to  improve 
consistency  across  university  programs  and  help  achieve  standardization  of  skill 
sets  for  graduates.  (Task  for  NDIA  SED  E&T  Committee.)3 

Consequently,  new  academic  and  career  pathways  are  urgently  needed  to  build  the 
talent  base  required.  Despite  an  increased  focus  on  science  and  engineering  education 
from  elementary  through  higher  education  since  the  launching  of  the  Sputnik,4  with  the 
number  of  undergraduate  and  graduate  systems  engineering  degree  programs  offered 
in  the  United  States  reaching  165  programs  across  80  institutions,5  the  best  methods 
and  approaches  for  developing  systems  engineering  curriculum  continue  to  evolve  as 
the  discipline  evolves. 6’7,8,9 

RT-19,  Research  on  Building  Education  &  Workforce  Capacity  in  Systems  Engineering 
(referred  to  as  the  SE  Capstone  Project)  is  expected  to  provide  a  substantial  addition  to 
this  knowledge  base.  This  research  is  being  conducted  in  the  context  of  14  “capstone” 
courses,  in  most  cases  as  an  integrative  culminating,  project-based  course  involving 
teams  of  students  working  together  on  the  development  of  a  product  or  prototype  that 
addresses  a  real  Department  of  Defense  need.  Implemented  as  pilot  courses  in  eight 
civilian  and  six  military  institutions  of  higher  education  (IHEs)  affiliated  with  the  Systems 
Engineering  Research  Center  (SERC),  these  14  institutions  are  piloting  methods, 
materials,  and  approaches  to  create  new  courses  or  enhance  existing  courses  to 
embed,  infuse,  and  augment  systems  engineering  knowledge,  as  defined  by  the 
SPRDE-SE/PSE  Competency  Model,  among  undergraduate  and  graduate  students. 
This  pilot  program  is  being  conducted  in  order  to  inform  the  development  of  a  national 
scale-up  effort  that  would  substantially  expand  the  number  and  capabilities  of 
universities  able  to  produce  systems  engineering  graduates.  It  is  anticipated  that  the 
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implementation  of  the  pilot  courses  will  lead  to  the  identification  of  strategies  for  student 
recruitment,  exemplary  course  materials,  assessment  instruments,  models  for  external 
mentorship,  and  other  lessons  that  may  be  deployed  to  accelerate  the  adoption  of 
effective  practices  and  materials  in  a  national  scale-up. 

Ill  Background 

In  the  past  decade,  in  an  effort  to  address  workforce  development  needs  in  systems 
engineering,  separate  competency  models  for  systems  engineering  have  been 
developed  by  individual  companies,  consortia,  and  professional  societies.  Descriptions 
and  comparisons  of  these  models  are  available  from  a  variety  of  sources. 10,11,12  Over 
the  last  few  years,  one  such  model  development  took  place  in  the  defense  acquisition 
community.  Subject  matter  experts  developed  and  validated  the  Systems  Planning, 
Research  Development,  and  Engineering  (SPRDE)  Systems  Engineering  (SE)  and 
Program  Systems  Engineer  (PSE)  competency  model,  known  as  the  SPRDE-SE/PSE. 
The  SPRDE-SE/PSE  competency  model  is  comprised  of  twenty-nine  areas  of 
competency  grouped  according  to  three  primary  “units  of  competence”  -  analytical, 
technical  management,  and  professional  -  as  shown  below.13  The  analytical  unit  covers 
thirteen  competencies  related  to  the  technical  base  for  cost  and  aspects  of  the  system 
life  cycle.  The  technical  management  unit  addresses  twelve  competencies  on  the 
technical  side  of  project  management,  and  the  professional  unit  covers  the  broader 
competencies  of  communication,  problem  solving,  systems  thinking  and  ethics. 

In  general,  competency  models  are  used  not  only  for  workforce  selection  and 
development,  but  also  for  educational  purposes.  In  particular,  systems  engineering 
competency  models  are  becoming  more  widely  used  in  support  of  systems  engineering 
workforce  development,  education,  and  training.14,15,16,17,18  This  research  therefore 
draws  upon  the  SPRDE-SE/PSE  competency  model  to  define  learning  objectives  for  a 
set  of  courses  being  implemented  across  the  14  piloting  universities. 


s 

PRDE-SE/PSE  Competency  Model 

Analytical 

(13) 

1 .  Technical  Basis  for  Cost 

2.  Modeling  and  Simulation 

3.  Safety  Assurance 

4.  Stakeholder  Requirements  Definition 
(Requirements  Development) 

5.  Requirements  Analysis  (Logical  Analysis) 

6.  Architectural  Design  (Design  Solution) 

7.  Implementation 

8.  Integration 

9.  Verification 

10.  Validation 

11.  Transition 

12  System  Assurance 

13.  Reliability,  Availability,  and  Maintainability 

Technical 

14.  Decision  Analysis 
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Management 

(12) 

15.  Technical  Planning 

16.  Technical  Assessment 

17.  Configuration  Management 

18.  Requirements  Management 

19.  Risk  Management 

20.  Technical  Data  Management 

21.  Interface  Management 

22.  Software  Engineering 

23.  Acquisition 

24.  Systems  Engineering  Leadership 

25.  System  of  Systems 

Professional 

(4) 

26.  Communications 

27.  Problem  Solving 

28.  Strategic  Thinking 

29.  Professional  Ethics 

The  14  pilot  universities  are  required  to  address  one  or  more  of  four  DoD  problem  areas 
and  to  produce  an  actual  product,  prototype,  or  other  artifact  to  demonstrate  their 
learning. 
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DoD  Problem  Areas 

1 .  Low-cost,  low-power  computers  leveraging  open-source  technologies  and 
advanced  security  to  support  sustainable,  secure  collaboration; 

Portable,  renewable  power  generation,  storage,  and  distribution  to  support 
sustained  operations  in  austere  environments  and  reduce  dependency  on  carbon- 
based  energy  sources; 

Portable,  low-power  water  purification; 

2.  An  expeditionary  assistance  kit  around  low-cost,  efficient,  and  sustainable 
prototypes  such  as  solar  cookers,  small  and  transportable  shelters,  deployable 
information  and  communication  technologies,  water  purifiers,  and  renewable 
energies.  These  materials  would  be  packaged  in  mission-specific  HA/DR  kits  for 
partner  nation  use; 

3.  Develop  modular,  scalable,  expeditionary  housing  systems  that  possess  "green" 
electric  power  and  water  generation,  waste  and  wastewater  disposal,  hygiene,  and 
food  service  capabilities.  Systems  should  be  designed  to  blend  in  to  natural/native 
surroundings  and  with  minimal  footprint; 

4.  Continued  investigation  and  exploration  into  the  realm  of  the  possible  with  respect 
to  “Immersive”  training  technologies.  Objective  is  to  flood  the  training  audience 
environment  with  the  same  STIMULI  that  one  would  experience  during  actual 
mission  execution.  Where  possible  full  sensory  overload  is  desired  much  the 
same  as  experienced  in  combat.  Specific  S&T  areas  for  development 

Virtual  Human.  Successful  modeling  of  emotions,  speech  patterns,  cultural 
behaviors,  dialogue  and  gestures. 

Universal  Language  Model.  The  ability  for  trainees  to  seamlessly  converse 
with  the  Virtual  Human. 

Virtual  Character  Grab  Controls.  The  ability  for  exercise  controllers  to 
assume  control  of  virtual  characters. 

Automated  Programming.  Cognitive  learning  models  and  the  ability  for 
exercise  controllers  to  adjust  virtual/live  simulations. 


The  program  is  being  implemented  in  three  sequential  phases  over  a  19-month  period. 
During  Phase  1 /Planning  and  Startup  (March  1,  2010-May  15,  2010),  the  research 
team,  with  participation  from  the  sponsor  agency: 

•  Developed  the  requirements  and  specifications,  timeline,  and  funding  limits  for 
the  SE  Capstone  Projects 

•  Developed  the  research  design  and  project  evaluation  plan 

•  Developed  the  common  assessments  required  of  all  projects  and  the  model  for 
projects 

•  Developed  and  issued  the  request  for  proposals  and  selection  process  (an 
independent  review  team  and  rubric)  for  selecting  participating  institutions 

•  Concluded  with  the  selection  of  eight  civilian  universities  and  six  service 
academies  with  SE  programs  that  would  participate  in  the  SE  Capstone  pilot 
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Phase  2/Pilot  Implementation  (May  15,  2010-June  30,  201 1)  focuses  on  the 
development,  implementation,  and  evaluation  of  the  courses  at  14  institutions. 
Implementation  ranges  from  a  one-semester  “broad  spectrum”  introduction  to  systems 
engineering  course  to  a  two-semester  senior  design  course,  to  a  supplemental 
independent-study  approach  that  augments  a  traditional  senior  design  course  with  SE 
modules.  During  this  phase,  the  institutions  have  recruited  student  teams;  organized  the 
course  structure  (materials  and  faculty  participation);  identified  and  initiated 
collaborations  with  DoD  and  industry  mentors  to  serve  in  a  variety  of  roles  with  student 
teams;  developed  and  administered  their  own  course  assessments  based  on  the 
learning  objectives  of  their  particular  courses;  and  are  delivering  the  courses,  the 
majority  of  which  will  continue  through  the  spring  semester. 

Phase  3/Analysis,  Recommendations  &  Dissemination  (July  1, 201 1  -  September  30, 
201 1 )  will  analyze  the  results  from  all  Capstone  Team  Members  and  integrate  them  into 
a  single  set  of  observations  to  the  sponsor  about  the  effectiveness  of  the  pilot  programs, 
analysis  of  pre-/post  learning  of  SE  content,  skills,  and  career  interest,  and  the  degree 
to  which  learning  outcomes  were  achieved.  It  will  also  create  specific  recommendations 
on  how  to  scale-up  the  pilot  program  to  be  conducted  across  the  U.S.  A  key  vehicle  to 
synthesize  findings  across  the  various  projects  is  a  workshop  for  all  SE  Capstone  Team 
projects,  the  government  sponsor,  representatives  of  industry,  relevant  professional 
societies  and  accrediting  organizations  (including  ABET),  and  other  interested  parties  in 
July  201 1 .  At  that  workshop,  the  results  of  the  individual  SE  Capstone  pilots  will  be 
presented. 

As  part  of  the  pilot,  each  university  team  is  required  to  develop  its  own  customized  pre- 
and  post-assessments  tailored  to  the  specific  learning  objectives  in  the  courses  it  is 
developing.  Since  these  are  all  different,  each  university  team  is  also  required  to 
administer  three  common  assessments  to  all  participating  students.  These  will  be  used 
to  gauge  student  learning  of  specific  concepts  and  skills  related  to  systems  engineering 
and  will  allow  comparisons  across  universities,  as  well  as  provide  data  for  correlating 
different  outcomes  with  differences  in  implementation. 

During  the  first  months  of  the  project,  the  research  team  met  several  times  to  lay  out  the 
specific  objectives  for  student  learning.  The  final  objectives  for  students  were  that,  at 
the  end  of  the  capstone  course,  participating  students  should: 

•  Understand  what  systems  engineering  is 

•  Understand  what  systems  engineers  do 

•  Understand  the  qualities  and  skills  that  systems  engineers  bring  to  projects 

•  Consider  a  career  in  systems  engineering 

•  Develop  and  practice  the  skills  of  systems  engineers 

•  Understand  what  systems  engineers  do/how  systems  engineers  think  (analytic 
skills) 
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This  report  will  summarize  progress  as  of  January  201 1 .  It  is  based  on  an  analysis  of 
each  partner  institution’s  interim  report  and  the  student  baseline  (pre-course)  survey.  A 
final  section  includes  preliminary  observations  and  recommendations. 

IV  DoD  Problem  Area  Addressed 


Each  university  chose  one  or  more  problem  areas  based  on  existing  faculty  expertise 
and  interest.  Two  civilian  universities  and  one  service  academy  chose  to  work  in  more 
than  one  of  the  DoD  problem  areas.  More  than  half  the  projects  (8)  addressed  DoD 
Problem  Area  1;  Problem  Area  4,  with  the  remaining  two  areas  divided  among  the  other 
partner  institutions: 


DoD  Problem  Areas  Addressed 


60% 

50% 

40% 

30% 

20% 

10% 

0% 


Problem  Area  1:  Low- 
cost,  low-power 
computers 


Problem  Area  4: 
Immersive  training 
technologies 


Problem  Area  2: 
Expeditionary 
assistance  kit 


Problem  Area  3: 
Expeditionary 
housing  systems 


■  Seriesl 


57.10% 


28.60% 


21.40% 
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The  following  table  shows  problem  area(s)  address  by  institution: 


Auburn  University 

1 

Missouri  S&T  University 

4 

Penn  State  University 

2 

Southern  Methodist  University 

4 

Stevens  Institute  of  Technology 

3 

University  of  Maryland 

1 

University  of  Virginia 

1,4 

Wayne  State  University 

1,3 

Air  Force  Institute  of  Technology 

1 

Naval  Postgraduate  School 

3 

US  Air  Force  Academy 

1 

US  Naval  Academy 

1,2 

li 


US  Military  Academy 

4 

US  Coast  Guard  Academy 

1 

V  Structure  and  Organization  of  SE  Capstone  Courses 

Each  university  organized  the  structure  and  content  of  its  pilot  course(s)  differently, 
depending  primarily  on  the  existing  structure  of  the  capstone  courses  within  the 
institution,  the  expertise  and  interests  of  the  faculty  recruited  into  the  effort,  the  types  of 
students,  and  the  mentors  available.  The  key  features  that  differentiated  the 
organizational  structure  of  the  programs  at  the  different  IHEs  were  the  following: 

•  Faculty:  The  collaboration  of  two  or  multiple  faculty  members  on  course  design  and 
implementation 

•  Courses:  The  integration  of  the  SE  component  into  existing  courses  or  the  creation 
of  entirely  new  courses 

•  Course  sequencing:  The  implementation  of  a  course  sequence  that  included  an 
introductory  course  followed  by  a  capstone  experience  or  a  capstone  experience 
only 

•  Student  population:  The  involvement  of  either  undergraduates  and  graduate 
students  as  learners*  or  only  one  of  these 

•  Mentors:  The  presence  and  level  of  active  and  meaningful  involvement  of  DoD  and 
industry  mentors  in  a  variety  of  student  learning  experiences 

The  following  sections  will  discuss  these  in  more  detail. 

VI  Faculty  Involvement 

A  majority  of  the  universities  relied  on  the  expertise  of  systems  engineering  faculty  to 
lead  or  contribute  to  the  conceptualization,  development,  and  implementation  of  the 
program,  but  many  other  faculty  were  involved  as  well,  particularly  from  mechanical 
engineering  and  computer  science.  At  1 1  institutions,  faculty  came  from  at  least  three 
separate  engineering  disciplines,  literally  embodying  the  multi-disciplinarity  of  a  systems 
engineering  team.  In  the  following  graph,  percentages  represent  the  percentage  of  the 
14  pilot  universities  that  included  those  types  of  disciplinary  faculty  in  the  RT-19  project: 


+Some  institutions  engaged  graduate  students  as  teaching  or  technical  assistants,  but  here  we  are  referring  to 
graduate  students  who  are  participating  in  the  course  for  credit. 
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Faculty  Involvement 
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Nearly  two-thirds  of  the  fourteen  projects  were  planned  and  implemented  by  teams  of 
two  or  three  faculty  members,  but  four  projects  included  four  or  more  faculty.  Only  one 
institution  developed  a  capstone  course  that  was  planned  and  taught  by  a  single  faculty 
member: 


Faculty  teams 


■  2  to  3  faculty  members 
per  institution 

■  4  or  more 


□  1  faculty  member 


Faculty  took  on  different  roles,  including  that  of  classroom  instructor,  curriculum 
developer,  project  advisor,  and  SE  subject  matter  expert,  with  some  being  several  of 
these.  In  this  graph,  percentages  represent  the  percent  of  all  faculty  in  the  project  where 
faculty  could  play  more  than  one  role. 
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All  Pis  reported  that  their  universities  had  been  very  supportive  of  the  effort,  with  this 
support  coming  in  a  variety  of  forms: 
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VII  Course  Sequences,  Structures,  and  Types  of  Student 

There  was  a  diverse  array  of  methods,  approaches  and  structures  for  the 
implementation  of  the  courses.  The  table  on  the  following  page  summarizes  the 
differences  in  type  of  student  (graduate/undergraduate/both),  course  integration,  and 
type  of  course  sequence.  (More  detailed  information  for  each  institution  can  be  found  in 
Appendix  1.) 
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Students: 

Undergraduates/ 

Graduate 

Integrated  into 
Existing  SE 
Courses 

Intro  course  + 
Capstone/ 
Capstone 
only 

Auburn  University 

U 

N 

l/C 

Missouri  S&T  University 

U/G 

Y 

C 

Penn  State  University 

U 

Y 

C 

Southern  Methodist  University 

U 

Y 

l/C 

Stevens  Institute  of  Technology 

U/G 

Y 

C 

University  of  Maryland 

U 

Y 

C 

University  of  Virginia 

U 

Y 

C 

Wayne  State  University 

U/G 

Y 

C 

Air  Force  Institute  of 

Technology 

G 

Y 

l/C 

Naval  Postgraduate  School 

G 

Y 

l/C 

US  Air  Force  Academy 

U 

Y 

C 

US  Naval  Academy 

U 

Y 

C 

US  Military  Academy 

U 

Y 

C 

US  Coast  Guard  Academy 

U 

Y 

C 

All  but  one  institution  integrated  the  RT-19  effort  into  existing  SE  courses,  with  only 
Auburn  developing  a  completely  new  course.  Thirteen  of  the  14  projects  were  structured 
into  two  semesters,  with  only  one  institution  (Penn  State)  conducting  a  single-semester 
capstone  project  course.  However,  one  (University  of  Maryland)  is  conducting  two  one- 
semester  capstone  courses  (i.e.,  with  different  students).  Those  who  carried  over  two 
semesters  did  so  in  one  of  two  ways:  (1 )  The  first  semester  was  an  introduction  to 
fundamental  SE  concepts  and  processes  and  the  second  was  devoted  to  the 
development  of  the  capstone  design  project  or  (2)  both  semesters  were  devoted  to  the 
capstone  project  work.  Appendix  2  lists  the  course  materials,  course  management 
systems,  and  software  applications  developed  or  used  by  each  institution. 

The  student  population  also  varied  in  terms  of  the  mix  of  graduate  students  and 
undergraduates.  Nine  of  the  institutions  had  only  undergraduates  as  students,  two  had 
exclusively  graduate  students  and  three  had  a  mix.  In  addition,  two  institutions  had 
graduate  students  acting  as  TAs  or  providing  technical  help.  (See  Section  IX  below  for 
more  on  the  student  population.) 

Student  teams  ranged  from  four  to  seven  members.  Teams  met  during  class,  at  lab 
sessions  at  some  institutions,  and  also  communicated  through  a  number  of  non-face-to- 
face  channels,  including  e-mail,  telephone,  videoconference,  weblogs,  and  on 
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collaborative  document  sharing  platforms.  Teams  generally  submitted  weekly  progress 
reports  and  prepared  final  project  presentations.  At  two  institutions,  student  teams 
included  distance  students  who  communicated  with  their  teams  via  their  university 
Blackboard  portal  or  the  Stevens  project-wide  Sakai  site.  In  half  the  universities, 
students  chose  their  own  teams;  in  the  rest,  faculty  did  so: 


Formation  of  Student  Teams 


■  Student  self-selection 


■  Faculty  selection-  with 
students  matched  by  GPA 
or  other  factors 

■  The  student  group  was 
small  and  there  was  only 
one  team 

■  Faculty  selection  -  with 
some  student  input 
(based  on  problem  area) 


Over  75  percent  of  all  capstone  student  teams  were  mixed  in  discipline,  allowing  faculty 
and  students  to  experience  working  in  a  context  resembling  that  of  professional  systems 
engineers: 
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VIII  Student  Recruitment 


The  nature  of  the  funding  cycle  resulted  in  a  very  compressed  period  in  which  to  recruit 
students  for  the  SE  Capstone  project.The  timeline  varied  across  the  different  IHEs, 
according  to  the  schedule  of  the  start  of  classes,  which  ranged  from  as  early  as  mid- 
August  in  some  institutions  to  early  September  in  others.  Moreover,  a  majority  of 
students  selected  the  next  year’s  courses  in  the  prior  term  and  have  either  no 
availability  or  flexibility  in  their  schedules  to  make  changes  if  a  new  course  is 
announced  late. 

Those  universities  that  already  had  existing  capstone  design  courses  in  place  were  able 
to  infuse  DoD  projects  into  them  and  were  able  to  recruit  students  who  had  not  yet 
committed  to  a  specific  project.  For  recruiting,  most  institutions  relied  primarily  on  face- 
to-face  interactions  with  potential  students  and  recommendations  from  other  faculty 
members,  and  most  relied  on  more  than  one  avenue: 
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One  PI  noted  that  the  presence  of  two  DoD  mentors  at  the  initial  recruitment  session 
“was  a  key  factor  in  the  recruitment  of  the  students  ...  as  it  allowed  students  to  ask 
questions  and  get  a  real  feel  for  the  purpose  of  the  projects.”  Other  reasons  for 
successful  student  recruitment  included  “the  humanitarian  nature  of  the  projects”  or 
recommendations  by  student  advisors. 

On  a  scale  of  1  to  5,  the  short  time  frame  and  resulting  course  scheduling  issues  were 
seen  by  the  Pis  as  the  most  challenging  factors  in  recruiting  students,  while 
administrative  support  and  student  interest  were  the  least  challenging  factors: 
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Least 

challenging 

(1) 

(2) 

(3) 

(4) 

Most 

challenging 

(5) 

Average 

Short  time  frame 

1 

0 

1 

3 

6 

4.18 

Course  scheduling  issues 

4 

0 

0 

3 

5 

3.42 

Lack  of  student  interest 

2 

0 

3 

1 

1 

2.86 

Lack  of  faculty  support 

6 

1 

3 

0 

0 

1.70 

Lack  of  administrative 
support 

6 

1 

1 

0 

0 

1.38 

IX  Analysis  of  Student  Participants 

This  section  will  provide  an  overview  of  the  demographic  characteristics  of  the  students 
engaged  in  the  RT-19  capstone  courses,  based  on  analysis  of  the  student  pre-surveys 
from  13  of  the  institutions  that  submitted  them.*  Although  there  were  differences  among 
the  institutions,  this  analysis  will  focus  on  the  entire  population. 

Underqraduates/Graduate  students 

The  total  number  of  students  returning  surveys  was  294.  Of  those,  127  were 
undergraduates,  124  were  graduate  students,  and  11  were  postgraduates. 

While  the  total  number  of  undergraduates  and  graduate  students  was  nearly  equal 
across  the  13  institutions,  a  closer  look  at  differences  between  individual  institutions 
shows  that  nearly  half  of  the  13  institutions  (Penn  State,  UVA,  SMU,  CGA,  AFA,  and 
West  Point)  were  comprised  entirely  of  undergraduates.  Four  institutions  (Wayne  State, 
AFIT,  NPS,  and  the  Naval  Academy)  had  graduate  students  (including  postgraduate 
students)  and  the  remaining  three  (Auburn,  Missouri  S&T,  and  Stevens)  had  mixed 
undergraduate  and  graduate  populations.  Flowever,  the  ratio  varied:  while  Auburn  had 
mostly  graduate  and  postgraduate  students  (92  percent),  with  fairly  few 
undergraduates,  Stevens  had  a  2:3  ratio  of  undergraduates  to  graduate  students  and 
Missouri  S&T  had  a  1:4  ratio. 


*  Baseline  data  is  missing  from  the  University  of  Maryland. 


18 


VO 

<N 

II 

c 


c 

cu 

"D 

3 

■M 

VO 

M— 

o 

cu 

-Q 

E 

3 


140 

120 

100 

80 

60 

40 


20 
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Most  of  the  undergraduates  were  in  their  senior  year  and  most  of  the  graduate  students 
were  in  their  first  year: 


All  Institutions  -  By  Class  year 
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H  Seriesl 

10 

113 

94 

28 

19 

Class  size 

Class  sizes  varied  widely  across  institutions,  ranging  from  a  low  of  3  or  4  students 
(AFIT,  USMA)  to  48  (NPS).  The  average  class  size  was  20  (median  =  17,  SD  =  14). 

Major 

The  most  prevalent  engineering  discipline  among  students  across  all  institutions  was 
Systems  Engineering,  followed  by  Mechanical  Engineering,  Electrical  Engineering, 
Computer  Science/Software  Engineering,  and  Industrial  Engineering. 
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Twelve  of  the  13  institutions  included  students  from  two  or  more  engineering  disciplines. 
In  one  institution,  Auburn,  students  came  from  as  many  as  five  different  engineering 
disciplines.  Only  one  institution  (CGA)  had  all  of  its  students  from  one  discipline, 
mechanical  engineering. 

The  most  prevalent  major  among  students  overall  was  Systems  Engineering,  followed 
by  Mechanical  and  Electrical  Engineering.  While  about  a  quarter  of  the  total  (n=264) 
population  returning  surveys  majored  in  Systems  Engineering,  these  students  were 
distributed  among  8  of  the  13  institutions.  Students  who  majored  in  Mechanical 
Engineering,  the  second  most  popular  engineering  discipline,  were  distributed  across  9 
institutions.  Electrical  Engineering,  the  third  most  popular  major,  was  distributed  over  six 
institutions.  All  Software  Engineering  and  87  percent  of  Computer  Science  major  were 
in  one  institution,  Auburn. 

Other  majors  represented  in  student  survey  responses  included  Product  Architecture; 
Alternative  Energy  Technology,  Chemical  Engineering,  and  Biomedical  Engineering. 
However,  the  percentage  of  students  majoring  in  these  subjects  across  institutions  was 
insignificant  and  they  were  often  based  in  specific  institutions. 
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Gender  and  Ethnicity 


All  Institutions  -  By  Gender 
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Overall,  the  student  population  (n=264)  was  over  three-quarters  male  (77  percent),  with 
a  small  female  population  (16  percent)  and  a  small  percentage  of  students  selecting  not 
to  report  their  gender  (7  percent). 

Students’  reported  ethnicities  (n=264)  broke  down  into  White  (64  percent);  Asian  (1 1 
percent);  Black  or  African-American  (7  percent);  Hispanic/Latino  (5  percent);  Hawaiian 
or  Other  Pacific  Islander  (>1  percent);  and  unreported  (11  percent). 


All  Institutions  -  By  Ethnicity 
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Experience  with  Engineering 


Students  from  12  of  the  13  institutions  reported  having  engineering  internship  or 
cooperative  experience.  However,  this  varied  greatly  by  institution,  from  a  low  of  12 
percent  of  the  students  at  one  university  to  a  high  of  69.2  percent  at  another.  Overall, 
students  had  the  most  engineering  work  experience  during  the  summer,  followed  by  full¬ 
time  engineering  employment,  coop  work,  and  work  during  the  academic  year.  Students 
in  over  half  of  the  institutions  had  engineering  experience  in  three  or  more  of  these 
activities. 


All  Institutions  -  By  Engineering  Experience 
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Experience  with  Systems  Engineering 

Despite  over  half  of  the  student  population  having  some  form  of  engineering  experience 
and  the  fact  that  there  were  Systems  Engineering  majors  in  many  institutions, 
approximately  40  percent  of  the  students  who  returned  surveys  reported  no  experience 
with  Systems  Engineering  and  another  20  percent  reported  that  they  did  not  know  if 
they  had  any  such  experience — for  a  total  of  60  percent  of  the  total  population.  These 
students  were  distributed  across  11  of  the  13  institutions. 
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All  Institutions  -  By  Systems  Engineering  Experience 
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Of  those  students  who  did  report  having  systems  engineering  experience  (and  the 
survey  allowed  them  to  check  off  multiple  forms  of  work  experience),  25  percent  had 
taken  coursework,  12.5  percent  had  had  full-time  employment  in  systems  engineering,  7 
percent  had  a  coop  or  internship  experience;  and  1  percent  had  systems  engineering 
work  experience  during  the  institution  year. 

Interest  in  Systems  Engineering  as  a  Career 

Over  half  (52  percent)  of  all  students  reported  a  level  of  high  interest  in  becoming  a 
systems  engineer,  18  percent  report  moderate  interest,  and  1  percent  reported  little 
interest. 


All  Institutions  -  SE  Career  Interest 
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Working  for  the  government  as  a  systems  engineer 


Forty-two  percent  of  the  students  reported  that  they  might  want  to  work  for  the 
government  as  a  systems  engineer;  24  percent  said  they  had  little  interest  in  doing  so, 
and  21  percent  said  they  had  interest  (21%).  1 1  percent  were  not  sure. 


All  Institutions  -  Insterest  in  working  for 
the  government  as  systems  engineer 
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Working  in  private  sector  as  a  systems  engineer 

Forty-five  percent  of  students  reported  wanting  to  work  in  systems  engineering  but  in 
the  private  sector,  while  20  percent  showed  only  moderate  interest  in  this  and  19 
percent  expressed  little  interest. 


All  Institutions  -  Interest  in  working  for 
the  private  sector  as  systems  engineer 
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Students  without  expressed  interest  in  systems  engineering 


There  was  a  small  population  of  students  (8  percent  of  all  students)  who  were  “Not  at  all 
interested”  in  pursuing  systems  engineering  careers,  either  in  the  government  or  private 
industry.  Some  of  their  reasons  included  wanting  to  work  as  an  engineer,  but  not  as  a 
systems  engineer,  in  their  particular  branch  of  the  military;  attending  business  or 
graduate  education  in  their  engineering  discipline;  or  planning  to  work  for  themselves  as 
business  consultants  or  entrepreneurs. 

Career  interests  of  Systems  Engineering  majors 

Of  the  subpopulation  of  Systems  Engineering  majors  (n=65),over  half  expressed  high 
interest  (4  and  5  on  a  scale  of  1  to  5)  in  becoming  systems  engineers.  These  students 
reported  a  similarly  high  interest  (4  and  5  on  a  scale  of  1  to  5)  in  working  for  the 
government  or  private  industry  as  a  systems  engineer. 

Students  with  high  interest  in  systems  engineering  careers  in  government  and  the 
private  sector 

Finally,  there  was  a  subpopulation  of  students  (n=25),  or  about  1 0  percent  of  the  entire 
student  population  who  answered  “highly  interested”  to  all  three  questions.  These 
students  came  from  ten  different  institutions,  including  five  service  academies  and  five 
civilian  universities,  and  five  different  disciplines  (Systems  Engineering;  Alternative 
Energy  Technology;  Industrial  and  Software  Engineering).  Their  post-graduation  plans 
included  “going  to  department  head  school”;  working  in  private  industry  as  a  systems 
engineer;  working  in  their  particular  branch  of  the  military  as  a  systems  engineer; 
working  in  alternative  energy;  or  pursuing  systems  engineering  education  at  the 
graduate  or  doctoral  levels.  Sixteen  of  these  25  students  were  graduate  students,  6 
were  undergraduates,  2  were  postgraduates,  and  one  identified  as  a  PhD  doctoral 
student. 

X  Student  Outcomes  and  Assessments 

As  noted  above,  both  project-created  and  institution-created  assessments  are  being 
used  to  assess  student  outcomes.  The  project  assessments  were  developed  by  the 
research  team  and  are  being  administered  to  all  students  in  all  courses  in  all 
universities.  The  internal  assessments  were  developed  by  each  university  to  assess  its 
particular  course  or  capstone  experience. 

The  first  external  (or  common)  assessment  instrument,  administered  before  the 
students  began  any  coursework,  was  the  survey  that  gathered  information  about  the 
students’  backgrounds,  interests,  and  prior  experience  with  systems  engineering,  asked 
them  about  their  career  aspirations,  and  then  asked  them  five  open-ended  questions 
designed  to  assess  their  understanding  of  what  systems  engineering  is  and  of  systems 
engineering  careers.  The  post-course  survey  includes  several  satisfaction  questions 
and  then  repeats  the  same  five  open-ended  questions. 

The  second  external  assessment  has  been  designed  to  see  the  extent  to  which 
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students  have  integrated  systems  engineering  concepts  into  their  thinking  by  asking 
them  to  transfer  that  thinking  to  another  situation.  Before  they  began  any  substantive 
course  work,  students  were  presented  with  a  summary  of  the  research  and 
development  history  of  the  Bradley  Fighting  Vehicle  platform,  along  with  a  few  short 
clips  from  The  Pentagon  Wars — a  fictionalized  film  version  of  the  book  by  the  same 
name — to  engage  students  during  the  additional  time  commitment  needed  for  this  type 
of  assessment.  Students  were  asked  to  read  and  discuss  the  scenario  and  then 
respond  individually  to  a  single  prompt:  “Could  the  problems  encountered  in  developing 
the  Bradley  Fighting  Vehicle  have  been  avoided?  Explain  your  answer.  ” 

Faculty  were  provided  with  a  link  to  the  results  and  were  encouraged  to  use  these  to 
begin  a  discussion  of  the  role  of  systems  engineers  in  large-scale  complex  projects 
such  as  they  were  about  to  embark  upon.  The  same  material  is  then  presented  as  a 
post-test,  with  the  expectation  that  the  responses  will  be  more  detailed  and  will  show 
greater  evidence  of  systems  engineering  thinking  after  the  SE  Capstone  experience. 

The  following  table  details  the  external  assessments  completed,  and  the  internal 
assessments  developed,  by  institution: 


RT-19  COMMON  EXTERNAL 
ASSESSMENTS 

INTERNAL 

ASSESSMENTS 

Using  blogging  tool 

Pre¬ 

survey 

Pre-case 

study 

Auburn 

University 

Yes 

[student's  journal 
w/reflections] 

Yes 

Yes 

Homework  assignments, 
case  study,  mid-term 
exam,  final  exam. 

Missouri 

S&T 

Yes 

[PhD  students  and 
mentors'  reflections 
on  students' 
progress] 

Yes 

Yes 

Presentations  performed 
for  mock  reviews. 
Components  of  the  final 
written  project  document 
used  to  evaluate  progress. 

Penn  State 

Yes 

[student's  journal 
w/reflections] 

Yes 

Yes 

Pre/post  surveys 
(additional  questions). 

SMU 

Yes 

[student's  journal 
w/reflections] 

Yes 

Yes 

Course  assignments. 

Stevens 

Institute 

One  entry  as  a  test 

Yes 

Yes 

Peer  assessments. 

Rubric  developed  for 
overall  project  will  be 
applied  in  Spring. 

UMD 

Yes 

[students’  journals 
w/reflections] 

Not  yet 

Not  yet 

Lab  assignments. 

Instructors'  observations. 
Final  project  presentation. 
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UVA 

Yes  [UVA's  Sakai 
Platform] 

[student's  journal 
w/reflections] 

Yes 

Yes 

Presentations,  team 
reports. 

Design  Process  Knowledge 
Critique 

Wayne 

State 

Yes 

[Wayne  State's 
website] 

[student's  journal 
w/reflections] 

Yes 

Yes 

SE  Student  Query, 

Interview,  and  Response 
Tool  [SE-SQUIRT]. 

Team  designs--final  report. 

AFIT 

Not  yet 

Yes 

Yes 

Class  assessment 
associated  with 
introductory  SE  course. 
Presentations  [e.g.  design 
reviews]. 

Final  thesis  project. 

NPS 

Not  yet 

Yes 

Yes 

Currently  developing 
formative  and  summative 
assessment  instruments 
related  to  SE  learning. 

Naval 

Academy 

Yes 

[Pi's  journal  on 
project's  progress] 

Not  yet 

Not  yet 

The  only  assessment  was 
that  contained  within  the 

DAU  courses. 

Coast 

Guard 

Academy 

Yes 

[posts  of  documents 
developed  for 
project] 

Yes 

Yes 

Assignments  such  as 
requirements  specification, 
preliminary  design  reports, 
oral  progress  reports. 

Military 

Academy 

Not  yet 

Yes 

Yes 

Initial  client  meeting,  Initial 
progress  reviews,  client 
out-briefing,  peer 
assessment,  interim  tech 
report. 

Air  Force 
Academy 

Yes 

[student's  journal 
w/reflections  and 
posts  of  documents 
developed  for 
project] 

Yes 

Yes 

The  majority  of  the  grading 
elements  were  taken  from 
three  team  presentations, 
[guests  from  industry  and 
outside  faculty  were  invited 
to  critique  the  teams' 
presentations]. 

At  the  time  RT-19  interim  reports  were  submitted  in  January  2011,  no  university  had 
completed  an  analysis  of  student  learning  outcomes  resulting  from  its  course(s). 
Anecdotal  information,  in  the  form  of  student  blog  posts  and  comments  from  external 
mentors,  suggest  that  the  SE  Capstone  courses  are  impacting  student  learning,  interest 
in  DoD  problems,  and  future  SE  education  and  careers. 
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Excerpts  of  such  anecdotal  evidence,  below,  are  included  only  to  provide  an  illustration 
of  the  types  of  impacts  the  SE  Capstone  courses  are  having.  They  are  not  intended  to 
provide  evidence  of  a  pattern  or  of  project-wide  impact. 

Excerpts  from  student  blog  posts 

Week  1 

“This  week  I  met  and  introduced  myself  to  my  team  members  and  we  learned  some  of 
the  principles  of  systems  engineering.  This  was  done  by  in  class  lectures  and  by 
examining  a  case  study  of  the  Bradley  Fighting  Vehicle.  ” 

“The  systems  engineering  competencies  that  we  learned  and  applied  this  week  were: 
stakeholders’  requirements,  definition,  requirements  analysis,  and  technical  planning. 

In  class  we  did  an  exercise  where  we  used  stakeholders’  requirement  definition  and 
requirements  analysis.  In  the  exercise  we  looked  at  examples  of  what  a  stakeholder 
requirement  would  be.  Then  the  requirement  was  interpreted  to  exactly  what  it  might 
mean  in  engineering  terms.  The  case  study  of  the  Bradley  Fighting  Vehicle  showed  the 
things  that  could  happen  if  there  is  poor  technical  planning.  ” 

Week  1 

“This  week  was  just  plain  awesome.  Our  team  was  able  to  finalize  our  Risk  Mitigation 
plan  which  was  pretty  helpful  in  identifying  some  areas  of  our  system  that  we  hadn't 
necessarily  considered  before.  When  discussing  some  of  the  questions  we  had  with  the 
Comms  team  as  well  as  [our  mentor],  we  found  that  pursuing  a  WiFi  network  instead  of 
a  cell  network  would  be  the  more  viable  option  for  conceptual  purposes,  with  the 
understand  that  a  fully  implemented  system  would  need  a  more  powerful  and  capable 
network  such  as  micro-cell.  ” 

“Some  of  the  systems  engineering  areas  that  we  focused  on  this  past  week  included 
things  like  Risk  management,  architecture  design,  and  decision  analysis.  They  were 
helpful  in  the  fact  that  they  put  value  to  how  engineers  should  go  about  weighing  the 
pros  and  cons  of  risks  in  determining  how  vital  they  are  in  relation  to  other  risks  that 
may  arise  when  solving  problems  of  a  system.  ” 

Week  12 

a.  What  did  you  and  your  team  accomplish  this  week? 

“This  week,  we  ordered  our  printer/scanner  and  external  hard  drive  for  the  LSA  system. 
We  also  revised  our  schedule,  and  finished  our  critical  design  report.  We  present  our 
critical  design,  as  well,  to  the  faculty,  and  they  gave  us  feedback  and  made  us  think 
about  certain  areas  problems  could  arise  and  what  we  should  do  about  them.  ” 

b.  Which  systems  engineering  competencies  best  align  with  what  you  did  this  week? 
“Interface  Management,  Professional  Ethics” 
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c.  What  specifically  did  you  do  in  terms  of  each  of  the  competencies  you  checked? 
“Interface  Management  -  we  have  begun  thinking  about  and  verifying  how  our  interfaces 
will  align  with  that  of  the  Comms  team,  since  we  are  using  their  network.  It  was 
determined  that  we  will  be  able  to  communicate  voice  information  of  their  system. 
Professional  Ethics  -  In  class,  we  talked  about  standards  and  codes,  and  how  ethical 
situations  can  occur  in  the  workplace.  ” 

Final  Blog 

“I  feel  that  the  things  I  learned  about  systems  engineering  had  a  large  impact  on  my 
project.  I  was  not  aware  of  the  amount  of  types  of  documentation  that  a  systems 
engineering  project  required.  The  different  competencies  like  requirements 
management  and  verification  and  validation  showed  how  important  organizational 
aspects  are  to  a  successful  project.  I  think  they  resulted  in  a  higher  quality  project.  ” 

“I  think  systems  engineering  is  different  than  the  design  process  I  was  used  to  because 
systems  engineering  focuses  more  making  sure  that  the  system  accomplishes  what  is 
required  of  it.  The  design  process  I  was  used  to  often  consisted  of  simply 
accomplishing  the  goal  by  any  means.  ” 

“The  advantage  of  a  multidisciplinary  team  is  that  we  can  pull  our  knowledge  from  our 
disciplines  to  advance  our  project.  However,  because  the  project  got  focused  down,  one 
group  member  was  relied  upon  more  heavily  than  the  others.  The  most  difficult  task  to 
execute  was  the  risk  mitigation  plan  because  nobody  in  the  group  was  certain  about  the 
actual  risks.  ” 

“A  systems  engineering  course  would  be  good  for  someone  completing  a  project 
because  it  introduces  several  different  design  considerations  like  risk  and  requirements. 
This  will  enhance  the  design  of  the  project.  ” 

“Systems  engineering  would  be  a  good  career  because  of  the  potential  to  work  on 
several  different  types  of  projects,  not  just  the  projects  in  your  discipline.  You  will  also 
get  to  work  with  several  different  types  of  engineers.  ” 

DoD  Mentor  Feedback: 


The  following  is  an  email  from  a  DoD  mentor  on  impressions  of  course  impacts  on  the 
students: 

“Our  first  semester  with  Penn  State  was  a  great  success!  We  sponsored  four  projects, 
one  within  each  of  the  Prepositioned  Expeditionary  Assistance  Kit  capability  areas 
(water  purification,  power  generation,  local  situational  awareness  and  communications). 
Each  of  the  teams  produced  a  functioning  prototype  that  met  or  exceeded  the  objectives 
established  at  the  beginning  of  the  semester.  I  recently  attended  the  Fall  Engineering 
Showcase  where  each  of  the  projects  was  displayed  and  evaluated.  The  Local 
Situational  Awareness  and  Communications  teams  were  recognized  as  the  top  two 
projects  in  the  Systems  Engineering  category.  Seventeen  students  participated  in  the 
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DDRE  sponsored  projects.  At  least  two  have  expressed  interest  in  pursuing  SE  Masters 
degrees  this  next  year  and  more  in  the  future.  Several  of  the  students  are  graduating 
and  three  will  be  entering  the  workforce  in  Defense  related  positions.  Also,  PSU  has 
asked  that  we  sponsor  projects  again  in  the  Spring  Semester.” 

Myself  and  Mr.  Phil  Stockdale, Technical  Manager  for  the  PEAK  JCTD,  are  reviewing 
the  project  deliverables  with  our  SMEs  and  will  incorporate  them  into  the  PEAK  JCTD  or 
other  projects  as  applicable.  The  objective  of  the  PEAK  JCTD  is  to  demonstrate  and 
transition  an  array  of  capabilities  that  can  be  pre-positioned  to  help  provide  essential 
services  in  time  sensitive  events.  PEAK  will  provide  these  services  to  local  responders 
and  the  local  populace  in  support  of  a  wide  variety  of  missions,  to  include  support  for 
humanitarian  assistance/disaster  relief  (HA/DR)  activities. 

COL  Nancy  Grandy 

DDR&E/Rapid  Fielding  Directorate 

Mentor  to  Penn  State  University  Capstone  Team 


XI  DoD  and  Industry  Mentors 

Nine  of  the  Pis  reported  that  they  had  DoD  mentors,  but  four  of  the  five  who  did  not 
were  military  institutions.  Six  of  the  Pis  reported  that  they  had  industry  mentors,  but  only 
five  institutions  reported  that  they  had  both  as  of  the  end  of  the  fall  semester.  However, 
many  of  these  were  pre-existing  relationships  and  the  mentor  process  was  uneven  in  its 
degree  of  effectiveness,  as  the  following  sections  describe. 

The  following  table  includes  information  by  institution: 


DoD 

Industry 

Auburn  University 

N 

N 

No  DoD  or  industry  mentors  during  fall 
semester;  plans  for  recruiting  both  DoD  and 
industry  mentors  during  spring  semester 

Missouri  S&T 

University 

Y 

Y 

1  DoD  mentor,  Pete  Mueller,  from  Office  of 
Naval  Research;  5  industry  mentors 
(systems  engineers)  from  the  Boeing 
Company 

Penn  State  University 

Y 

N 

2  DoD  mentors  (Col.  Nancy  Grandy, 
Oversight  Executive  (Joint  Logistics)  Office 
of  Deputy  Undersecretary  of  Defense  Rapid 
Fielding  Directory;  Phil  Stockdale  of 

National  Defense  University);  no  industry 
mentor  but  student  teams  used  a  university- 
employed  collaborator  who  formerly 
worked  at  Honeywell  International 
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Southern  Methodist 
University 

Y 

N 

1  DoD  mentor  from  Marine  Corp.  Base 
served  in  a  recruitment  capacity;  4  DoD 
clients  from  Marine  Corp  base  visited 
institution  to  deliver  specs  for  project;  no 
industry  mentor 

Stevens  Institute  of 
Technology 

Y 

N 

Number  of  DoD  mentors  not  specified;  no 
industry  mentors 

University  of  Maryland 

Y 

Y 

DoD  mentors  at  Aberdeen  Proving  Ground 
and  Army  Research  Lab  (number  not 
specified);  1  industry  mentor,  Sandy 
Friendenthal,  from  Lockheed  Martin 

University  of  Virginia 

Y 

N 

1  DoD  mentor,  Bill  Campbell,  used  as  SE 
subject  matter  expert;  2  DoD  clients  for 
water  quality  testing  team:  Col  Nancy 

Grandy,  Oversight  Executive  (Joint 

Logistics)  Office  of  Deupty  Undersecretary 
of  Defense  Rapid  Fielding  Directory;  Phil 
Stockdale  of  National  Defense  University;  1 
DoD  client  for  medical  training  simulator 
team,  Dr.  Alan  Liu  of  the  National  Capital 
Area  Medical  Simulation  Center  (SlmCen) 
of  the  Uniformed  Services  University  of  the 
Health  Sciences  (USUHS) ;  no  industry 
mentors 

Wayne  State  University 

Y 

Y 

2  DoD  mentors,  Ms.  Claudia  J.  Quigley  of 
Natick  Labs,  USA  AMC,  &  Dr.  Peter  Schihl, 
Chief  Technologist-Mobility  at  US  Army 
TARDEC;  3  industry  mentors  from 
Mechanical  Engineering  Industrial  Advisory 
Board  (IAC) 

AFIT 

Y 

Y 

Air  Force  Research  Lab  subject  matter 
experts  used  as  DoD  mentors  (no  number 
specified);  industry  mentors  from 

Cooperative  Engineering  Services,  Inc. 
(CESI)  &  CLMax,  a  UAV  design  and 
fabrication  business  (number  not  specified) 

NPS 

N 

N 

No  DoD  or  industry  mentors  during  fall 
semester;  plans  for  recruiting  both  DoD  and 
industry  mentors  during  spring  semester 

USAFA 

N 

Y 

No  DoD  mentors;  recently  established  ties 
with  industry  mentors  from  two  companies: 
Greenvolts  (solar  industry)  and  Schweitzer 
Electronics  (power  industry) 
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USNA 

N 

N 

No  DoD  or  industry  mentors 

USMA 

Y 

Y 

1  DoD  mentor/client,  LTC  Joe  Nolan; 
recently  established  ties  to  industry  mentor 
Sarnoff 

USCGA 

N 

N 

No  DoD  or  industry  mentors 

DoD  Mentors 


Even  Pis  who  had  DoD  mentors  reported  very  different  levels  of  interaction  with  them. 
Despite  efforts  on  the  part  of  the  DoD,  several  had  not  been  assigned  or  connected  with 
projects  at  the  time  of  submission  of  the  interim  reports.  Thus  while  over  46  percent  of 
the  Pis  reported  that  their  DoD  mentors  were  “very  involved,”  another  46  percent 
reported  not  having  a  mentor  or  not  yet  working  with  their  mentor: 


Involvement  of  DoD  Mentors 


Very  involved 


Somewhat  involved 


We  have  a  mentor  but  we 
have  not  worked  with 
him/her  yet 

We  do  not  have  a  mentor 
yet/No  response  (1) 


Some  Pis  reported  that  their  mentors  were  playing  an  “active  role  in  meeting... by 
phone,  email  and  VTC,”  while  others  reported  that  the  mentor  had  not  followed  up  with 
the  students  after  some  initial  communication.  Over  30  percent  of  the  Pis  reported  that 
DoD  mentors  communicated  several  times  a  semester  with  students,  and  36.4  percent 
reported  that  they  did  so  on  a  weekly  or  biweekly  basis.  One  PI  even  reported  having 
daily  interaction: 
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Frequency  of  DoD  mentors-students  interactions 


40% 

35% 

30% 

25% 

20% 

15% 

10% 

5% 


u/o 

Daily 

Weekly 

Biweekly 

A  few  times  during 
the  semester 

Never 

HSeriesl 

28.60% 

14.30% 

14.30% 

28.60% 

35.70% 

Like  RT-19  faculty,  DoD  mentors  facilitated  student  learning  in  a  variety  of  ways; 
however,  their  roles  differed  instructionally  depending  on  whether  they  also  served  as 
clients,  as  they  did  for  50  percent  of  participating  institutions.  Some  of  their  roles 
included: 

•  Providing  feedback  on  student  projects  and  deliverables 

•  Providing  technical  expertise 

•  Attending  student  presentations 

•  Facilitating  field  trips  offsite  to  manufacturing  or  design  sites  where  students 
could  observe  engineering  processes  related  to  their  projects 


Mentors  with  the  highest  level  of  student  engagement  interacted  with  students  in  every 
single  activity  area  and  with  frequency,  while  others  provided  only  intermittent 
correspondence.  Note  jhat  mentors  could  engage  in  more  than  one  kind  ofjnteraction: 


DoD  Mentor-Student  Interactions 


60% 


50% 

40% 

30% 

20% 

10% 

0% 


Correspondence 
Classroom  via 

visit(s)  teleconference/ 

videoconference 


E-mail 


Correspondence 
via  telephone 


Offsite  visit(s) 


No  interaction 
(non-applicable 
because  of  lack 
of  mentor) 


HSeriesl 


50% 


28.60% 


21.40% 


7.10% 


7.10% 


35.70% 
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The  Pis  therefore  differed  in  their  evaluation  of  the  mentors’  contributions  to  student 
learning  and  engagement.  Note  that  mentors  could  play  make  contributions  in  more 
than  one  area: 


Contribution  of  DoD  Mentors 


50% 

45% 

40% 

35% 

30% 

25% 

20% 

15% 

10% 

5% 

0% 


Motivation 
for  students 


Feedback  on 
student 
projects 


Technical 

expertise 


Client 

requirement 
s  (the 
mentor  is 
also  the 
client) 


Initial  on¬ 
site  visit  to 
campus 


Access  to 
sites 


DoD  mentor 
has  not 
been  easily 
accessible  to 
students 
and  did  not 
provide 
support  or 
advice  of 
great  value 


DoD  mentor 
initially 
motivated 
students  but 
did  not 
follow  up 
with  further 
consistent 
communicat 
ion 


No 

engagement 
(non- 
applicable 
because  of 
lack  of 
mentor) 


MSeriesl 


42.90% 


42.90% 


42.90% 


35.70% 


21.40% 


14.30% 


7.10% 


7.10% 


35.70% 


One  PI  reported  that  their  DoD  expert  was  “outstanding,”  having  developed  a  seminar 
for  students  and  worked  with  them  on  prototype  evaluations.  Others  stated  that  their 
mentors  demonstrated  up-to-date  technologies  in  the  students’  problem  area;  offered 
“invaluable  comments  and  direction”  for  student  projects;  and  played  an  “active  role”  by 
monitoring  project  progress  through  multiple  communication  channels. 


However,  Pis  from  two  institutions  described  their  relationship  with  the  DoD  mentors  as 
“unsuccessful”  due  to  communication  breakdowns  brought  on  by  personnel  changes  on 
the  mentors’  end  and  another  wrote  that  although  the  DoD  mentor  had  provided  initial 
motivation  for  students  by  visiting  the  classroom,  there  had  not  been  much 
communication  as  the  semester  progressed  because  of  the  mentor’s  distance  and 
time/scheduling  constraints. 


Industry  Mentors 


About  60  percent  of  the  Pis  reported  not  having  an  industry  mentor.  Three  reported  that 
they  planned  to  work  with  an  industry  mentor  during  the  spring  semester. 
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Industry  mentors  took  on  roles  similar  to  the  roles  played  by  DoD  mentors — as  clients, 
consultants,  or  SMEs.  Note  that  mentors  could  play  more  than  one  role: 


Industry  Mentors'  Roles 


60% 

50% 

40% 

30% 

20% 

10% 


U70 

Gave  feedback  and 

technical  advice 

Provided  technical 
or  equipment 
support 

Served  as  a  client 

No  industry  mentor 

y  Seriesl 

42.90% 

14.30% 

7.10% 

57.10% 

Pis  at  two  institutions  that  had  both  industry  and  DoD  mentors  reported  that  having  both 
types  of  mentor  benefited  the  students’  projects.  For  example,  one  PI  reported  that  the 
DoD  mentor  acted  as  a  client  while  also  offering  occasional  help  with  analysis  and 
understanding  system  requirements,  while  the  industry  mentor  acted  as  a  consultant 
“educating  [students]  on  their  technical  approach.”  The  other  PI  whose  team  included 
both  types  of  mentors  described  the  industry  mentor  as  “part  of  the  project 
teams... intimately  involved  with  [students’]  day  to  day  progress  and  project 
management  aspect,”  compared  with  the  DoD  mentor,  who  took  the  customer’s 
perspective  and  dealt  more  with  design  and  engineering  concerns. 
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At  another  institution,  however,  a  PI  expressed  dissatisfaction  with  the  mentor’s 
overlapping  roles  of  client  and  project  advisor/consultant  and  argued  for  better  defined 
responsibilities.  Initially  regarded  by  students  as  a  “customer”  or  “major  stakeholder,” 
the  DoD  mentor  functioned  in  reality  as  a  subject  matter  expert  who  assisted  students 
with  the  design  process. 

XII  Effectiveness  of  Course  Design-Preliminary  Observations 

The  ultimate  measure  of  effectiveness  of  the  course  design  will  be  determined  by  the 
student  outcomes  identified  as  project  goals: 

•  Increase  student  learning  of  SE  competencies 

•  Increase  student  interest  in  DoD  problems/careers 

•  Increase  career  interest  in  SE  study  and  careers 

It  is  premature  at  this  stage  of  course  implementation  to  correlate  student  outcomes 
with  the  structure  or  content  of  the  courses  or  with  any  particular  strategies  or  course 
materials.  However,  the  Pis’  interim  reports  provide  some  insights  into  preliminary 
lessons  learned  about  course  objectives  and  implementation.  These  observations  fall 
into  several  categories: 

1 .  Challenges  teaching  the  broad  topic  of  systems  engineering  to  non-SE  majors 
under  time  constraints. 

2.  Challenges  with  equivalent  grading  policies  in  multi-disciplinary  teams, 
particularly  where  SE  was  an  overlay  to  an  existing  multi-disciplinary  team 
structure. 

3.  Challenges  with  content-domain-specific  problem  areas  and  with  finding 
meaningful  ways  for  other  disciplinary  majors  to  contribute. 

4.  Motivating  external  mentors  to  bring  authentic  professional  experiences  to  the 
learning  experience. 

5.  SE  content  modules  provide  opportunities  to  bring  non-SE  majors  to  a  common 
understanding.  These  have  been  implemented  with  varying  frequency,  durations, 
and  numbers  across  several  projects. 

6.  Where  possible,  the  integration  of  DAU  modules  provide  additional  incentives  for 
students  to  gain  desirable  certifications,  but  scheduling  common  experiences  for 
students  to  take  these  modules  impacted  completion. 

And  in  some  cases,  the  challenges  identified  were  common  to  the  formation  and 
operation  of  effective,  multi-disciplinary  teams: 

7.  Provide  subject  matter  expertise  (internal  and  external)  to  infuse  sufficient 
disciplinary  knowledge  such  that  students  may  focus  on  the  bigger  SE 
competencies. 
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Excerpt  of  Pilot  University  Interim  Report 


“The  biggest  challenge  early  in  the  course  was  getting  the  groups  formed  and 
communications  established  while  mixing  both  distance  and  local  students.  Unlike  other 
undergraduate  courses,  the  design  project  for  this  course  not  only  involved  a  high  level 
of  detail  but  also  included  an  interdisciplinary  approach.  This  made  some  of  the 
students  uncomfortable  with  the  deliverables  of  the  project.  This  was  overcome  by 
providing  subject  matter  experts  from  both  the  assisting  faculty  and  industry  to  assist 
the  groups  in  areas  that  their  members  did  not  have  expertise.  In  this  way,  the  students 
were  able  to  focus  more  on  the  systems  engineering  approach  and  less  on  the  details  of 
their  particular  engineering  disciplines.  ” 

It  is  also  not  clear  at  this  stage  the  extent  to  which  RT-19  funding  has  created  entirely 
new  materials  or  simply  (and  in  some  cases,  substantially)  enhanced  existing  courses. 
This  is  an  area  for  further  investigation.  Several  institutions  indicate  that  this  funding  has 
provided  the  capital  to  design  a  coherent  course  sequence  or  to  build  additional 
courses,  but  more  information  is  needed  to  measure  such  impacts  across  the  range  of 
participating  institutions. 

XIII  Allocation  of  Resources 

At  just  beyond  the  mid-point  of  project  implementation,  it  is  noteworthy  that,  as  at  Dec. 

31 , 2010,  only  30%  of  subcontractor  budgets  have  been  expended.  Preliminary 
analysis  points  to  several  possible  explanations: 

1 .  Lag  time  in  billing  for  expenses 

2.  Lag  time  resulting  from  delayed  awarding  of  Phase  II  and  the  challenges  involved 
with  Phase  II  being  awarded  in  two  segments  (a  and  b).  Phase  II  a  was  awarded 
May  1 5,  201 1  (with  a  completion  date  of  July  31 , 201 0)  and  Phase  II  b  was 
awarded  June  25,  2010.  The  slow  release  of  the  funding  to  the  collaborator 
universities  proved  challenging  administratively  to  these  institutions  resulting  in  a 
late  start  for  billing. 

a.  With  the  exception  of  one  collaborator  university,  invoicing  against  the 
project  started  August  2010  while  actual  work  had  been  started  May  when 
their  subcontract  was  issued. 

3.  Time  estimated  for  faculty  effort  for  course  development  during  summer  2010 
could  not  be  charged  to  RT-19  budgets  due  to  other  project  commitments 

4.  Summer  2010  labor  that  was  used  to  develop  the  coursework  was  not  charged  to 
the  project;  the  labor  budget  was  reallocated  to  the  spring  semester  (Jan.  -  May 
201 1 )  when  the  more  significant  portion  of  the  work  will  be  done. 

Upon  examination,  of  the  total  subcontractor  budgets,  the  following  represents  the 
expenditure  categories: 
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■  Faculty  -  39% 

■  Student  &  Tuition  -20% 


Travel  -  3% 


■  Materials  &  Supplies  =  7% 

■  F&A  -  32% 


While  the  original  level  of  funding  per  institution  (approximately  $200,000  per  institution) 
is  on  par  with  curriculum  and  course  development  funding  levels  from  other  agencies5, 
this  is  an  area  that  will  be  analyzed  further  when  the  final  project  report  is  submitted. 

Analyses  of  student  impacts  (degree  of  growth  on  pre-/post  measures  of  learning  and 
SE  career  interest  and  knowledge/interest  of  DoD  problems)  across  the  projects,  as  well 
as  the  numbers  of  students  impacted  are  also  factors  that  should  be  used  to  assess  the 
appropriate  allocation  of  financial  resources,  as  some  institutions  impacted  as  many  as 
54  students,  while  others  impacted  as  few  as  five. 

XIV  Dissemination 

To  date,  the  following  dissemination  has  taken  place: 

•  A  presentation  on  RT-19  was  made  at  the  Annual  SERC  Research  Review  in 
College  Park,  Maryland  on  November  10,  2010. 

•  Six  draft  papers  have  been  submitted  to  the  American  Society  of  Engineering 
Education  (ASEE)  201 1  Annual  Conference  in  June  201 1 ,  as  follows: 


§  At  the  time  the  budget  recommendations  were  made  for  RT-19  the  National  Science  Foundation  Course, 
Curriculum,  and  Laboratory  Improvement  (CCLI)  program  awards  grants  with  a  ceiling  of  approximately  $200,000 
for  course  development  efforts. 
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# 

201 1  ASEE  -  SECC  Abstract  Title 

Authors/Institution 

725 

SE  Capstone:  A  Pilot  Study  of  14  Universities  to  Explore 
Systems  Engineering  Learning  and  Career  Interest  through 
Department  of  Defense  Problems 

SERC/RT-19 

Research  Team 

1077 

SE  CAPSTONE:  Introduction  of  Systems  Engineering  into 
an  Undergraduate  Multidisciplinary  Capstone  Course 

Penn  State 
University 

1206 

SE  CAPSTONE:  Implementing  a  Systems  Engineering 
Framework  for  Multidisciplinary  Capstone  Design 

Stevens  Institute 
of  Technology 

1211 

SE  Capstone:  Integrating  Systems  Engineering 
Fundamentals  to  Engineering  Capstone  Projects: 

Experiential  and  Active 

Wayne  State 
University 

2186 

SE  CAPSTONE-  Introducing  Multidisciplinary  Design  to 
USCGA 

US  Coast  Guard 
Academy 

2669 

Fostering  Systems  Engineering  Education  through 

Graduate  Capstone  Projects 

Air  Force  Institute 
of  Technology 

Further,  SE  Capstone  partner  universities  are  encouraging  the  submission  of  student 
posters  to  two  April  201 1  events: 

•  April  12-14,  2011:  12th  Annual  Science  &  Engineering  Technology  Conference/DoD 
Tech  Exposition,  North  Charleston,  SC 

•  April  29,  201 1 :  201 1  IEEE  Systems  and  Information  Engineering  Design 
Symposium,  Charlottesville,  VA 

XV  Recommendations 

At  this  stage  of  project  implementation,  there  are  several  prevalent  themes  that  have 
been  identified  to  improve  the  course  implementation.  These  have  been  culled  from 
interim  reports  submitted  by  Pis  and  observations  by  the  research  team  include: 

1 .  More  planning  time  is  needed  to  effectively  plan  and  coordinate  course  materials 
and  assessments,  the  availability  of  external  resources  such  as  DoD  and  industry 
mentors,  as  well  as  facilities  visits,  and  to  secure  buy-in  and  define  roles  and 
responsibilities  of  interdisciplinary  faculty  participants. 

2.  In  order  to  effectively  recruit  students,  the  specific  DoD  problem  area  should  be 
defined  and  disseminated  to  students  at  least  at  the  time  students  register  for  their 
next  cycle  of  courses. 
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3.  Better  connections,  more  clearly  defined  roles,  and  stronger  commitment  by  DoD 
and  industry  mentors  would  enhance  students’  experiences  where  the  participation 
of  external  mentors  has  been  lacking.  Consideration  of  a  nominal  financial 
commitment  by  external  clients  may  increase  the  investment/commitment  by  these 
mentors. 

4.  A  list  of  required  and  recommended  faculty  events  and  student  programs  should  be 
made  available  to  Pis  to  encourage  maximum  participation  and  sufficient  financial 
resources. 

XVI  Future  Reports  and  Future  Research 

As  indicated  throughout  this  document,  the  preliminary  observations  and  findings 
represent  only  a  snapshot  of  the  richness  of  the  14  SE  Capstone  courses  that  have 
been  and  are  being  implemented  by  partner  institutions.  One  significant  limitation  of 
this  report  is  that  no  comprehensive  analysis  of  student  learning  has  been  conducted  at 
this  stage  of  course  implementation.  The  final  report  will  aim  to  connect  the  course 
content  and  organization,  including  materials  created  by  faculty  as  well  as  the 
contributions  of  external  mentors,  with  impacts  on  student  learning  of  SE  content,  their 
interest  in  SE  careers,  and  their  interest  in  DoD  problem  areas  and  careers. 

As  noted  in  the  introductory  section  of  this  report,  this  research  has  been  undertaken  in 
order  to  inform  the  development  of  a  larger  capacity-building  and  scale-up  effort  that 
could  substantially  increase  the  SE  workforce  available  to  DoD  and  industry  in  the  next 
decade  and  beyond.  RT-19  will  capture  methods,  strategies,  and  tools  that  have  led  to 
desired  student  outcomes.  Further  research  to  translate  these  findings  into  methods, 
tools,  and  processes  that  can  be  operationalized  in  new  universities  has  been 
proposed. 
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Appendix  1 :  Detailed  List  of  Course  Descriptions,  Problem  Areas,  and  Types  of  Student  by  Institution 


Institution 

Course/Project 

Description 

DoD  Problem  Area(s)/Focus 

Students 

Auburn 

Systems 

1st  course  [Fall  2010]  is  a  broad- 

Problem  Area  1 

3  U 

University 

Engineering 

spectrum  overview  to  systems 

Improvement  of  computer 

30  G 

in  a  Secure 

engineering.  It  introduces  major 

systems  to  enable  secure  data 

Computing 

concepts  using  a  case  study  of  the 

sharing  among  complex 

17  persist 

Intensive 

security  architecture  of  two  open 

systems  at  low  cost. 

to  2nd 

Environment 

systems  under  consideration  by 

DoD. 

Course  material  for  the  1st 

course 

course  will  be  delivered 

Includes: 

2nd  course  [Spring  2011]  is  an 

through  presentations  by 

CS,  IE, 

actual  project  employing  low-cost, 

speakers  from  industry  and 

and  EE 

open-source, 

government;  lectures,  and 

secure  computing.  The  students 

interactive  student  activities. 

On-campus 

will  demonstrate  secure 

The  2nd  course  is  a  hands  on 

and 

collaboration  using  the  Android 

distance 

open  source  software  stack. 

sequel  in  which  students  will 
complete  their  defense- 
focused  capstone  project. 

education 

Missouri  S&T 

Agile  Systems 

1st  Course  [Fall2010]:  Introduction 

Problem  Area  4 

10  U 

University 

Engineering- 

to  Systems  Engineering  provides 

Immersive  Training 

19  G 

Active 

the  student  with  basic 

Technologies. 

And  Experiential 

understanding  of  main  concepts, 

Subtle  simulation  of  real 

Includes 

Learning 

tools,  and  processes  of  systems 

battlefield  scenarios. 

ECE,  ME, 

Approach 

engineering. 

2nd  Course  [Spring201 1]:  Physical 

Operational  scenarios 
simulate  getting  shot,  getting 

and  AE 

hit,  and  minor  restriction. 

On-campus 

Artifact  Creation  and  Validation. 

and 

Development  of  detailed  design  for 

distance 

a  wireless  haptic  vest  with 
embedded  sensors.  Students  will 
focus  on  the  wireless  tech  to 

education 
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activate  embedded  sensors  and 
mechanical  components. 

Penn  State 
University 

Interdisciplinary 
Capstone  Design 
Project 

This  is  a  one-semester 
course/project  [Fall2010].  Eight 
modules  delivered  by  systems 
engineering  faculty. 

Projects  are  completed  using  the 
Bernard  M.  Gordon  Learning 

Factory,  a  lab  providing  modern 
design,  prototyping,  and 
manufacturing  facilities. 

Problem  Area  2 

Expeditionary  Assistance  Kit. 

1 .  Water  purification 
system 

2.  Power  generation  from 
renewable  energy 
sources 

3.  Local  situational 
awareness  system 

4.  Global  low-bandwidth 
communication  unit 

17  U 

Includes 

BE,  CE, 

EE,  ME,  IE 

Southern 

Methodist 

University 

Leveraging 

Interdisciplinary 

Teaching 

Environments  to 

Research 

Immersive 

Training 

Environments 

1st  Course  [Fall2010],  students 
work  in  interdisciplinary  teams  to 
design  an  architecture  solution  that 
meets  customer  specifications. 
Winter  break:  10  days  Skunk 

Works  Immersion  Design 

Experience.  (IDE) 

2nd  Course  [Spring  2011],  students 
will  continue  to  work  on 
interdisciplinary  teams  to  build  and 
test  a  prototype  of  their  design. 

Problem  Area  4 

Immersive  Training. 

The  objective  is  to  improve 
existing  capabilities  in  three 
areas:  (1 )  fidelity  of  motion 
capture  systems,  (2)  reduction 
of  infrastructure  required  for 
team  based  motion  capture, 
and  (3)  high  resolution  facial 
expression  capture  and 
replication 

11  U 

Includes 

CS,  EE 
and  ME 

3  PhD 
students 
serving  as 
teaching 
assistants 

Stevens 
Institute  of 
Technology 

Building 

Education  and 
Workforce 

Capacity  in 

Systems 

Engineering 

Implementation  of  SE  in  capstone 
senior  design  course  (two 
semesters):  a  series  of  6  Systems 
Engineering  all-day  workshops  are 
being  delivered  to  introduce  SE 
principles  and  methods  to  all 

Problem  Area  3 
Green-Expeditionary  Housing. 
For  a  100  person  FOB  and  3-6 
months  deployment.  Modular 
housing  with  micro-grid 
support  for  alternate  energy 

17  U 

7  G 

Includes 

EM,  ME, 

EE,  CE,  Civ 
Eng,  A&T 
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through  Capstone 
Design 

students.  Modifications  made  to 
adjust  to  scheduling  constraints. 
Design  component  of  capstone 
project  starts  in  Spring  201 1 . 

sources,  including  low  impact 
solutions  for  waste  and  water. 

University  of 
Maryland 

Special  Topics  in 

Systems 

Engineering 

This  is  a  one  semester  course  that 
is  offered  twice  over  one  academic 
year.  The  goal  of  this  pilot  is  to 
introduce  students  to  SE  thorugh  a 
hands-on  project  experience. 

Problem  Area  1 

Focuses  on  low-cost,  low- 
power  computers  leveraging 
open  source  technologies. 
Supports  integrated  wireless 
sensor  networks,  black  box 
design,  and  particle  steering 
using  micro  fluidic  devices. 

15  U  Fall;  4 
teams 

38  U  Spring 

Includes 

EE,  CE, 

BE. 

[assisted  by 

4  G] 

University  of 
Virginia 

Extensible 

Systems 

Engineering 

Capstone 

Experience 

Course  exposes  students  to  the 
entire  systems  engineering  process 
via  two  interdisciplinary  capstone 
projects  over  one  academic  year. 

During  the  2nd  semester  the  two 
teams  will  test  and  evaluate  each 
other’s  projects. 

Problem  Areas  1  and  4 

Project  #1  involves  a  virtual 
reality  system  for  medical 
training. 

Project  #2  is  focused  on 
developing  a  mobile, 
autonomous,  water  quality 
testing  system. 

17  U 

Includes 

SE,  ME, 

CS,  BE, 

ECE 

2  SE  G 

students 

provide 

technical 

support 
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Wayne  State 
University 

Integrated 

Material  Design 
and  Realization 
for  HA/DR  Kits 

This  project  integrates  SE  product 
development  concepts  across  4 
courses  at  the  undergraduate  and 
graduate  level. 

1  Full  semester  course  (Winter 

201 1 )  plus  modular  insertion  into 
multiple  other  courses  (start 
process  -  Sept  2010). 

Problem  Areas  land  3 
Expeditionary  Operations. 

The  projects  will  be  focused 
on  development  of  elements 
of  HA/DR  kits,  such  as  solar 
oven,  water  purification 
system,  alternative  energy. 

10  U 

19  G 

Includes 

ME,  ISE 

Air  Force 
Institute  of 
Technology 

Introduction  to 
Systems 
Engineering 
Process  and 

Design 

This  course  [one  academic  year] 
provides  a  broad  introduction  to  a 
systematic  approach  necessary  for 
the  formulation,  analysis,  design 
and  evaluation  of  complex 
systems.  Technical  support 
provided  by  the  Autonomous 
Navigation  Technology  Center 
associated  with  the  Department  of 
Electrical  and  Computer 

Engineering 

Problem  Area  1 

Low-power  computing  for 
operations  in  austere 
environments. 

Development  of  a  novel  hybrid 
electric  UAV  for  near  silent, 
long  loiter,  low  energy 
operations. 

5  G 

Mix  of  AE, 

SE 

Naval 

Postgraduate 

School 

Transforming 

Graduate 

Education  in 

Systems 

Engineering 

A  series  of  8  core  SE  courses  [one 
academic  year]in  the  masters 
curriculum  taught  via  team  based 
pedagogy,  with  the  capstone 
project  integrated  into  the  entire 
curriculum  as  a  carry  through, 
hands  on  experience.  The  courses 
provide  a  holistic  span  of  education 
from  systems  thinking,  , 
quantitative  analysis,  through 
system  design  and  production 

Problem  Area  3 

Expeditionary  Operations  and 
HA/DR  Assistance  Kits. 
Development  of  novel,  low 
density  power  supplies, 
advanced  materials  with  low 
thermal  and  visibility 
properties,  low  signature 
communication  devices. 

[Project  starts  in  January 

2011], 

48  G 

SE 
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US  Air  Force 
Academy 

Capstone  Design 
Project 

This  project  integrates  sequentially 
two  SE  courses  over  one  academic 
year.  Students  will  learn  to 
successfully  work  in  a 
multidisciplinary  team,  to  apply  SE 
and  management  tools, 
communicate  project  details,  and 
evaluate  contemporary  military 
issues. 

Problem  Area  1 

Low  Power  Computing 

A  10  KVA  solar  energy  system 
for  deployed  operations.  The 
end  system  will  incorporate 
smart  grid  technology  to 
facilitate  control  and 
integration 

[Project  starts  in  spring 
semester]. 

7  U 

Includes 

EE,  CE, 

ME,  SE 

US  Naval 
Academy 

Principles  of 
Engineering 
Systems  Design 

The  senior  design  capstone  course 
[one  academic  year]  is  enhanced 
with  additional  SE  sections  based 
on  experimental  coursework. 
Enhancements  include 
independent  study  course  based 
on  Defense  Acquisition  University 
courses  leading  to  certification 
opportunities. 

Problem  Areal  and  2 
Expeditionary  Ops. 

Portable,  low  power  water 
purification. 

Portable,  renewable  power 
generation,  storage  and 
distribution 

[most  of  the  project-centric 
work  is  done  in  the  spring 
semester.] 

16  U 

Includes 

EE,  CE, 

NA,  OE 

US  Military 
Academy 

Systems  and 
Engineering 
Management 
Design 

This  capstone  course  [two 
sequential  courses  over  one 
academic  year]  emphasizes  SE  in 
technology  based  organizations. 
Cadets  examine  interconnections 
between  planning,  organizing, 
leadership,  control,  and  the  human 
element  in  production,  research 
and  service  organizations 

Problem  Area  4 

Immersive  Training 

Augmented  Reality:  synthetic 
environ,  decision  analysis  for 
optical  &  video  displays,  high 
fidelity  tracking. 

4  U 

Includes 

SE,  EM, 
and  OR 

US  Coast 
Guard 

Academy 

Systems 

Engineering 

Capstone 

This  course  [one  academic  year] 
incorporates  critical  elements  of 

SE.  Cadets  will  have  regular 

Problem  Area  1 

Expeditionary  Ops.  Green 
Power  Generation  HA/DR 

24  U 

Includes 
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Enhancement 

contact  with  external  customers 

Portable  hull  inspection 

Civ  Eng, 

[USCG  Shore  Maintenance 

system. 

EE,  and  ME 

Command  and  USCG  Aviation 

Green  electric  power  in 

Logistics  Center]  and  must  defend 

remote  hot  climates. 

their  work  at  preliminary  and  final 

In  water  remote  propeller 

design  reviews 

cleaner. 

Hybridization  system  for  fleet 
vehicles. 

NOTES: 

1.  Sources  include  proposal  documents  submitted  by  universities,  interim  reports,  and  a  summary  prepared  by  R. 
McGahern  for  a  presentation  at  the  2010  Annual  SERC  Research  Review  conference  Nov.  9-10,  2010. 

2.  The  number  of  students  shown  in  the  table  above  include  only  those  who  are  directly  involved  in  the  whole 
capstone  experience  [coursework  +project]. 

3.  Abbreviations  :  U:  Undergraduates,  G:  Graduate  Students,  EM:  engineering  management,  CE:  computer 
engineering,  Civ  Eng:  civil  engineering,  EE:  electrical  engineering,  NA:  naval  architecture,  OE:  ocean  engineering, 
AE:  aerospace  engineering,  A&T:  arts  and  technology,  OR:  operations  research,  IE:  industrial  engineering,  ME: 
mechanical  engineering,  CS:  computer  science,  SE:  systems  engineering,  BE:  biomedical  engineering,  ECE: 
electrical  and  computing  engineering,  ISE:  Industrial  &  Systems  Engineering 
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Appendix  2:  Detailed  List  of  Course  Content  and  Materials 


Partner 

Institution 

Course  Materials 

Auburn 

University 

SE  Lecture  topics:  conceptual  design,  preliminary  design,  detail 
design,  testing,  open  source  computing  systems,  acquisition,  security 
certification,  systems  security,  decision  analysis,  configuration 
management,  economics,  real  world  systems  engineering. 

Standards:  National  Information  Assurance  Training  Standard  for 
Senior  Systems  Managers. 

Course  management  system:  Blackboard™ 

Software  applications:  l-CAIV  [decision  analysis],  Eclipse  &  Papirus 
[SysML  diagram] 

Archive  of  video  recorded  lectures  [for  students  viewing] 

Missouri  S&T 
University 

SE  Lecture  topics:  system  definition  and  concepts,  requirements  and 
specifications,  dynamic  object-oriented  requirements  system 
[DOORS]  presented  by  BOEING  mentor,  functional  analysis  and 
decomposition,  quality  function  deployment  [QFD],  conceptual 
systems  design,  DoD  architecture  framework,  risk 
identification/management,  Sys  Eng  planning  ,  architecture 
evaluation,  manufacturing  and  disposability,  supportability,  economic 
evaluation,  preliminary  design  review,  reliability,  system  test  and 
evaluation,  trade  off  studies,  modeling  and  simulation,  detail  design, 
optimization  in  design  and  operations,  writing  specifications. 

Standards:  PMT  90-S002K  Multiple  Integrated  Laser  Engagement 
System  [MILES]  Communication  Code  [MCC]  STANDARD.  MIL- 
STD-882D  DoD  Standard  Practice  for  System  Safety. 

Penn  State 
University 

SE  Lecture  topics:  systems  engineering  fundamentals,  systems 
requirements  analysis  and  allocation,  systems  architecture,  problem 
solving  in  system  design,  decision  and  risk  analysis,  introduction  to 
project  management,  systems  verification  and  validation,  introduction 
to  systems  thinking 

Reference  material:  NASA  Systems  Engineering  Handbook,  2007 
Course  management  system:  ANGEL  [Penn  State  Management 
System] 

Southern 

Methodist 

University 

SE  Lecture  topics:  overview  of  systems  engineering,  problem 
definition  and  requirements,  engineering  design  process,  incremental 
commitment  model  [ICM]. 

These  joint  SE  lecture  notes  were  complemented  with  specific 
separate  lectures  focused  on  CSE,  ME,  and  EE 

Course  management  system:  Blackboard™ 
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Stevens 

Institute  of 
Technology 

6  all  day  work-shops  to  introduce  students  to  key  SE  principles  and 
methods. 

WS-1:  Intro  to  the  project,  general  overview  of  the  SE  process,  and 
ConOps 

WS-2:  System  Level  Architecture 

WS-3:  Subsystem  Level  Architecture 

WS-4:  Logistics  and  Life  Cycle  Support 

WS-5:  Subsystem  Integration  and  Test 

WS-6:  System  Level  Integration  Test 

Course  Management  System:  Google  Groups,  Google  Docs, 

Dropbox 

SE  Software  applications:  Labview,  Matlab,  Solidworks 

University  of 
Maryland 

SE  Lecture  topics:  introduction  to  systems  engineering,  strategies  of 

SE  development,  foundations  for  model-based  systems  engineering, 
modeling  system  structure  and  system  behavior,  object  and 
component  based  development,  multi-objective  trade  studies, 
requirements  engineering,  systems  engineering  with  UML  and  SysML 
[Sandy  Friedenthal  from  Lockheed  Martin  delivered  a  special  lecture 
on  SysML,  which  was  recorded],  system  level  design,  basic 
approaches  to  system  validation/verification,  basic  approaches  to 
system  validation/verification. 

Course  management  system:  UMD’s  Institute  for  Systems  Research- 
Website 

SE  software  applications:  ParaMagic™  v16.6  spl,  Matlab/simulink 

University  of 
Virginia 

Project  1 :  SE  Design-Rapid  Adaptive  Needs  Assessment  [RANA] 

Unit  (1)  -  Problem  Definition,  Concept  of  Operations  Humanitarian 
Assistance  &  Disaster  Response  (HA/DR).  Unit  (2)  -  RANA  Kit  and 
Decision  Support  Tool  Requirements.  Unit(3)  -  Alternatives  and 
performance  criteria.  Unit  (4)  -  Building  and  testing.  Unit  (5)  -  Policy 
Context 

Project  2:  SE  Design-  Immersive  Virtual  Medical  Training  Capability. 
This  project  has  5  main  stages:  definition  of  the  problem,  drafting 
requirements,  prototyping  and  implementation  of  the  solution, 
integration,  and  validation  in  human-subjects. 

Course  Management  System:  UVa’s  Sakai  Platform 

Wayne  State 
University 

Lecture  topics:  introduction  to  systems  engineering,  concept 
evaluation  and  selection,  risk  management  in  design 

The  above  SE  principles  and  methods  are  complemented  with  the 
following  courses: 

1 .  Integrated  Product  Development  -  to  educate  students  about 
the  importance  of  concurrent  and  collaborative  engineering  in  a 
global  economy. 
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2.  Thermal-Fluid  System  Design  -  with  emphasis  on  alternative 
energy  tech 

Course  management  system:  Blackboard™ 

Air  Force 

Institute  of 
Technology 

Lectures  topics:  intro/process  overview,  conceptual  system  design 
and  requirements  definition,  model  based  SE  ,  utility  theory, 
preliminary  system  design,  detailed  design  and  development,  system 
test,  evaluation  &  validation,  reliability,  maintainability  &  supportability, 
affordability,  usability/human  system  integration.  These  topics  were 
complemented  by  3  case  studies 

Standards:  DoD5000,  JCIDS,  DAG 

Reference  Material:  INCOSE  Handbook 

Course  management  system:  Blackboard™ 

SE  software  applications:  Enterprise  Architect,  LEGO  Mind-storm 
robotics  kits 

Military 

Academy 
[West  Point] 

Lectures  topics:  systems  thinking,  stakeholder  analysis,  functional 
analysis,  value  modeling,  value  modeling  workshop,  in-progress 
review  [IPR],  modeling  and  simulation,  VBS2  Lab,  0*Net  Analysis, 
alternative  generation,  solution  enhancement. 

US  Air  Force 
Academy 

Lectures  topics:  Introduction  to  DFEC  Capstone  Design  Course, 
Requirements  development,  planning  and  scheduling,  functional 
analysis  and  allocation,  risk  management. 

Course  management  system:  MS-Sharepoint 

Software  applications:  MS-Project,  Crystal  Ball 

Naval 

Postgraduate 

School 

The  capstone  coursework  spans  almost  the  entire  core  curriculum  in 
the  SE  masters  degree  program.  The  RT-19  Project  is  fit  into  the  lab 
sections  of  the  following  courses:  Engineering  Project  Management, 
Systems  Suitability,  System  Architecting  and  Design,  Systems 
Integration  and  Development.  The  capstone  is  related  to  each  of  the 
other  core  courses  as  part  of  learning,  application,  and  assessment. 
Course  Management  System:  Sakai  [NPS  platform] 

Standards:  Naval  Educational  Skill  Requirements  [ESR],  specifically 
the  5800  ESR  for  systems  engineering;  ABET  for  SE;  the  CDIO 
initiative  standards 

SE  software  applications:  Vitech  CORE 

US  Coast 

Guard 

Academy 

Lectures  topics:  design  process  overview,  problem  definition  and 
need  identification,  quality  function  deployment,  concept  generation, 
functional  decomposition,  evaluation  [Pugh’s  matrix],  codes  and 
standards,  human  factors,  design  for  manufacture,  design  for 
assembly  &  recycling,  engineering  economics,  detail  design, 
engineering  ethics,  modeling  and  simulation,  risk-reliability-safety, 
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quality-robust  design-optimization 

Course  management  system:  Blackboard 

Software  Applications:  Solidworks 

Naval 

Academy 

The  schedule  is  based  on  completing  the  required  DAU  courses: 
Introduction  to  DoD  Science  and  Technology  Management, 
Fundamentals  of  Systems  Acquisition  Management,  Fundamentals  of 
Systems  Planning,  Research,  Development,  and  Engineering.  The 
discussions  focused  on  how  to  apply  the  topics  from  the  courses  to 
the  design  process  for  the  project  of  each  Senior  Design  Capstone 
Team 
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Appendix  3:  IRB  Status  as  of  January  2011 


University 

Local  IRB  approval 
granted  or  exempt 

Pl/Staff  with  IRB  consents 
received 

Auburn  University 

No 

David  Umphress,  Alice  Smith, 
C.W.  Perr 

Missouri  S&T 

Yes 

Cihan  Dagli 

Penn  State 

Yes 

James  Nemes,  Mary  Lynn 
Brannon,  Liz  Kisenwether, 
Kirsten  Hochstedt 

SMU 

Yes 

LiGuo  Huang,  Nathan  R. 
Huntoon 

Stevens  Institute 

Yes 

Keith  Sheppard,  Eirik  Hole 

UMD 

No 

Mark  Austin 

UVA 

Yes 

William  Sherer,  Reid  Bailey, 
Greg  Gerling,  Garrick  Louis 

Wayne  State 

Yes 

Darrin  Ellis,  Kyoung-Yun  Kim, 
Dean  Pichette,  Ming-Chia  Lai, 
Eugene  Rivin,  Yun  Seon  Kim 

AFIT 

Yes 

David  Jacques 

NPS 

No 

Dave  Olwel,  Clifford 

Whitcomb,  Ali  Rodger 

Naval  Academy 

Yes 

Not  yet 

Coast  Guard 

Academy 

No 

Richard  Freeman,  Ronald 
Adrezin,  Jonathan  Russell, 
Charles  Hatfield 

Military  Academy 

Yes 

Steve  Henderson 

Air  Force  Academy 

No 

Andrew  Laffely,  Alan  Mundy 
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